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The use of enhanced vision systems in civil aircraft is projected to increase rapidly as the Federal Aviation 
Administration recently changed the aircraft operating rules under Part 9 1 , revising the flight visibility requirements for 
conducting approach and landing operations. Operators conducting straight-in instrument approach procedures may now 
operate below the published approach minimums when using an approved enhanced flight vision system that shows the 
required visual references on the pilot’s Head-Up Display. An experiment was conducted to evaluate the 
complementary use of synthetic vision systems and enhanced vision system technologies, focusing on new techniques 
for integration and/or fusion of synthetic and enhanced vision technologies and crew resource management while 
operating under these newly adopted rules. Experimental results specific to flight crew response to non-normal events 
using the fused synthetic/enhanced vision system are presented. 


Introduction 

Synthetic vision (SV) is a computer- generated image of 
the external scene topography that is generated from 
aircraft attitude, high-precision navigation, and data of 
the terrain, obstacles, cultural features, and other 
required flight information. A synthetic vision system 
(SVS) enhances this basic functionality with real-time 
integrity checks to ensure the validity of the database, 
obstacle and navigation accuracy identification and 
verification, and continuous traffic surveillance 
functions. An Enhanced Vision (EV) System, EVS, 
(also referred to as an Enhanced Flight Vision System) 
is an electronic means to provide a display of the 
external scene by use of an imaging sensor, such as a 
Forward-Looking InfraRed (FLIR) or millimeter wave 
radar. The intended use of EV mirrors SV - both strive 
to eliminate low-visibility conditions as a causal factor 
in civil aircraft accidents and to replicate the operational 
benefits of clear day flight operations, regardless of the 
actual outside visibility condition. The methodology by 
which this capability is achieved through SV or EV, 
however, is significantly different. While some may 
consider the technologies to be competing; they are, in 
fact, complementary. 

SV, by virtue of being weather-independent and 
unlimited in field-of-regard, holds many advantages 
over enhanced vision sensor systems for providing 
terrain, path, and obstacle awareness, particularly during 
flight phases, such as approach, which may be obscured 
by clouds and precipitation that an EV sensor cannot 
penetrate. Recognition of terrain and cultural features 
may also be improved over an EV view since the 
display presentation is optimized by the display 


designer, not the product of the sensor and its 
environment. Pilot recognition of EV terrain and cultural 
features depends upon the reflected, emitted, and / or 
refracted energy at the spectral frequencies of the EV 
sensor and the ability of the pilot to (correctly) interpret 
this image. Atmospheric effects, time of day, and sensor 
characteristics can be important factors in the quality of 
the EV imagery. 

On the other hand, EV is an imaging sensor which 
provides a direct view of the vehicle external 
environment; consequently, EV is completely 
independent of the derived aircraft navigation solution 
and is independent of a database. Very little stands 
between the EV image shown to the pilot and the real- 
world; thus, an EV pilot gets an extremely high degree of 
confidence in the system. Under conditions of smoke, 
haze, and night, a FLIR/EV may provide orders-of- 
magnitude improvement over the pilot’s natural vision; 
greatly enhancing the pilot’s situation awareness and 
reducing the pilot’s workload. 

Previous synthetic vision research (Parrish et al., 2003) 
has shown that a “flight-critical” synthetic vision 
implementation which uses automated decision aiding 
functions for object detection and database 
alignment/navigation error detection produces superior 
performance to synthetic vision concepts with an EV 
inset display (e.g., McKay et al., 2002). To date, 
however, technology for “perfect” object detection and 
database/navigation error detection does not exist. 
Further, even if these systems come to fruition, there may 
still be gaps, such as minimal radar cross section objects 
or below- threshold detection values, which may require 
other additional integrity and error checks. SV with EV 
inset displays may offer one possible method to provide 
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the pilot with information sufficient to perform 
navigation integrity and obstacle clearance checks. 

While EV might improve SV operations, the converse 
warrants investigation as well. In 2004, Section §91.175 
of the Federal Aviation Regulations was amended such 
that operators conducting straight-in instrument 
approach procedures may now operate below the 
published Decision Height and Minimum Descent 
Altitude when using an approved Enhanced Flight 
Vision System (EFVS) on the pilot’s Head-Up Display 
(HUD). This rule change now provides “operational 
credit” for EV. As such, EV operations will become 
more prevalent. However, while EV may help with the 
“last 100 ft”, it may not supplement the pilot’s 
awareness of the terrain, obstacles, and flight path much 
outside of this area. SV may fill in this awareness 
“gap,” (i.e., before the EV sensor provides a useful 
image) and it may also provide the crew with “visual 
momentum” to assist the crew’s understanding and 
correct interpretation of the EV sensor imagery. 

An experiment was conducted to evaluate the 
complementary use of SV and EV technologies, 
specifically focusing on new techniques for integration 
and/or fusion of synthetic and enhanced vision 
technologies and crew resource management while 
operating under the newly adopted FAA rules which 
provide operating credit for EV. The overall objective 
of this experiment was to test the utility, acceptability, 
and usability of integrated/ fused enhanced and synthetic 
vision systems technology concepts in two-crew 
commercial and business aircraft cockpit for Required 
Navigation Procedures (RNP)-type approaches; these 
results are described in Bailey, Kramer, and Prinzel 
(2006). The current paper describes experimental results 
specific to flight crew response to non-normal events 
that were staged in this experiment using a fused 
synthetic/enhanced vision system. 

Methodology 

Subjects 

Twenty-four pilots, representing seven airlines and a 
major cargo carrier, participated in the experiment. All 
participants had previous experience flying HUDs. The 
subjects had an average of 1787 hours of HUD flying 
experience and an average of 13.8 years and 16.2 years 
of commercial and military flying experience, 
respectively. 


Simulator 

The experiment was conducted in the Integration Flight 
Deck (IFD) simulation facility at NASA Langley 
Research Center (LaRC). The IFD emulates a Boeing B- 
757-200 aircraft and provides researchers with a full- 
mission simulator capability. The oollimated out-the- 
window (OTW) scene is produced by an Evans and 
Sutherland ESIG 4530 graphics system providing 
approximately 200 degrees horizonlal by 40 degrees 
vertical field-of-view at 26 pixels per degree. Traditional 
primary flight and navigation displays were presented 
head-down. 

Head-Up Display 

The HUD subtended approximately 32° horizontal by 24° 
vertical field of view. The HUD presentation was written 
strictly in raster format from a video source (RS-343) 
input. The input consisted of a video mix of symbology 
and computer-generated scene imagery (either EV or SV 
as described in the following). The symbology included 
“haloing” to ensure the readability of tthe symbology 
against the background scene imagery background. 
Brightness and contrast controls were provided to the 
pilot. Also, the pilot had a declutter control, implemented 
as a push-button on the left hand horn of the Pilot Flying 
(PF) yoke. The button cycled through three “declutter” 
states: 1) No declutter (full symbology and scene 
imagery); 2) “Raster” declutter (full symbology, no scene 
imagery); and 3) “Full declutter” (no HUD display). 

Four HUD display concepts were tested, differing from 
each other in: 1) the type of raster background presented; 
and, 2) in the type of symbology presented. Two raster 
formats were flown, either EV only (hereinafter referred 
to as “FLIR”) or a fusion SV/EV image (hereinafter 
referred to as “Fusion.” 

The FLIR-only concept represented our “baseline” EFVS 
HUD condition. The Fusion HUD concept represents 
one method of providing complementary SV/EV 
information for the pilot flying. The Fusion HUD raster 
image started out as a pure SV image, transitioning 
through a fused SV/EV presentation beginning at 600 feet 
above ground level (AGL), and ending with a pure FLIR 
raster image by 500 feet AGL. Between 600 feet and 500 
feet AGL, the fusion gradually stepped from 100% SV / 
0% EV ending at 0% SV / 100% EV. Each raster 
concept showed FLIR below 500 ft to take advantage of 
the operational credit now offered by use of FLIR on the 
HUD. The 500 ft Fusion transition altitude was chosen 
from a prior usability study designed to assess an 
optimum altitude after which FLIR would be required. 

As mentioned earlier, two symbology sets were flown: 
Standard HUD symbology (hereafter referred to as 
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“Baseline”) and the same standard HUD symbology 
enhanced with pathway guidance and a runway outline 
(hereafter referred to as the “Tunnel” symbology set). 
The “tunnel” symbology set was tailored to transition at 
the same altitudes as the Fusion raster and this transition 
was based on flight test experience (Kramer et ah, 
2005). The tunnel was shown above the 500 ft Above 
Field Level (AFL) transition altitude, the last tunnel 
segment was positioned at 500 ft AFL (thus, it was no 
longer visible below 500 ft), and, upon reaching 500 ft 
AFL, the runway outline was drawn until 50 ft AFL. In 
Figures 1 and 2, two of the concepts are shown, the 
Fusion-Tunnel HUD and the FLIR-Baseline HUD. 



Figure 1. Fused HUD Concept with Tunnel 



Figure 2. Baseline HUD Concept (EV Only) 


Auxiliary Display 

The Pilot Not Flying- Auxiliary Display (PNF-AD) was 
located outboard of the PNF location which is not 
atypical of modem Boeing commercial aircraft 
electronic flight bag display installations. The 8.4” 
diagonal display was full-color with 1024 x 768 pixel 
resolution. The display video source was a video mix of 
symbology and computer-generated scene imagery. 

Four PNF-AD display concepts were tested, differing 
from each other in: 1) the type of raster background 


presented; and, 2) the type of symbology presented. The 
raster was either EV only (hereinafter referred to as 
FLIR) or a fused SV/EV image (hereinafter referred to as 
Fused). The symbology was either “On” or “Off’ for the 
data mns. When present, the symbology was a subset of 
the standard HUD symbology (Figures 3 & 4). 



Figure 3. EV Only - No Symbology 



Figure 4. Auxiliary Display Fusion - Symbology 


The AD fused raster image was pilot-controllable and 
could be tuned throughout the approach to one of 10 
states: FLIR only, SV only, or 8 fusion combinations of 
FLIR and SV, using an Equinox EP-3000™ fusion board. 
The fusion employs a feature-level extraction algorithm 
with two pilot control inputs: (1) feature-level fusion of 
FLIR and SV and (2) modulation of false-color coding of 
the fusion image. 

Synthetic Vision System 

A synthetic vision database was created from a 1 arc-sec 
(30 meter post- spacing) Digital Elevation Model (DEM) 
of a 53 x 53 nm area centered around the Reno-Tahoe 
International Airport (FAA identifier: KRNO). The 
airport was represented by three-dimension models of the 
mnway, taxiways, and terminal buildings. The DEM was 
draped with 1 meter/pixel satellite imagery within a 1 6 x 
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21 nm area centered around KRNO and 4 meter/pixel 
outside this inner region. 

Enhanced Vision System 

A physics-based FLIR simulation (using Evans & 
Sutherland EPX Sensors™) was created from the OTW 
visual database by applying materials properties to each 
component of the data. The characteristics of a 
short/mid-wave FLIR were simulated in a “white-hot” 
presentation. The time-of-day, time-of-year, and other 
diurnal properties were held constant. Atmospheric 
properties (cloud layer, cloud height and thickness, fog, 
and visibility) were varied experimentally to modulate 
the visibility that the evaluation pilots had in the FLIR 
and the OTW scene presentations. 

Evaluation Task 

The PF hand-flew the base and final leg portions of the 
Sparks Visual Arrival to RNO Runway 16R with 
autothrottles engaged at an approach speed of 138 knots. 
The PNF monitored the approach from the right-hand 
side of the flight deck using standard instruments and 
the AD. This high workload, curved descending 
approach (currently approved for visual conditions only) 
was chosen to evaluate the use of SV/EV technologies 
for performing an RNP-type approach while flying in 
restricted visibility. The aircraft was configured for 
landing prior to each run (landing gear down and flaps 
30 degrees). Pilot participants were instructed that the 
run would end at main gear touchdown but that they 
should perform a go-around if they felt the landing was 
not safe or if they did not have the required visual 
references to descend below decision height or to 
complete the landing as per FAR §91.175. 

Experiment Matrix 

Nominally, 42 experimental runs were completed by the 
evaluation crew with each pilot flying 21 approaches 
evaluating the HUD concepts and with each pilot 
monitoring 21 approaches while evaluating the AD 
concepts. The wind and weather varied on each run. 
The nominal visibility in the EV and OTW varied from 
1 mile down to U mile. The required EV visual 
references as per FAR §91.175 became visible on the 
HUD between 450 ft and 250 ft AFL, enabling the crew 
to descend to 100 ft Height Above Touchdown (HAT). 
4 runs per flight crew were specifically designed so the 
EV visual references were visible but the required 
runway (normal vision landing) references were not. 
These four runs, if properly flown using the EV crew 
procedures, should conclude by a go-around initiated no 
lower than 100 ft AFL. 


Six non-normal runs were also flown by each crew. The 
non-normals were runway incursion (RI) scenarios (2 per 
crew) and database integrity monitoring scenarios (4 per 
crew). The low number of RI and database integrity 
scenarios precluded expectancy on the part of the flight 
crew. The RI scenarios simulated an incursion with either 
a non-transponding baggage cart or fire truck. The 
database integrity monitoring scenarios purposefully 
introduced a lateral navigation solution error (of either 50 
or 75 feet) with respect to the real runway. This error 
resulted in the synthetic vision terrain, pathway and 
guidance cue being misaligned from the FLIR and ILS 
(which were defined in the flight crew briefing as always 
being correct). 

Procedure 

The subjects were given a 1-hour briefing to explain the 
SV/EV display concepts, EV crew procedures, and the 
expected evaluation tasks. After the briefing, a 2-hour 
training session in the IFD was conducted to familiarize 
the subjects with the aircraft handling qualities, display 
symbologies, SV/EV crew procedures, and controls. 
Special emphasis was placed on the 91.175 regulations 
pertaining to EV operations. The ‘rare-evenf scenarios 
were not discussed, although the pilot’s responsibility for 
maintaining safe operations at all times was stressed. 
Data collection lasted approximately 4.5 hours followed 
by a 30-minute semi-structured interview. The pilots 
were also given a final questionnaire. 

Results 

Nominal Run Results 

Experimental data was taken for flight technical error, 
mental workload, and situation awareness during all runs. 
The results are reported in Bailey, Kramer, and Prinzel 
(2005) and only the general findings are discussed below. 

Path Control Performance . Root-mean-square (RMS) 
lateral and vertical path error performance followed 
previous studies. The presence of a tunnel significantly 
enhanced flight technical performance during approach 
phase. The performance differences may not be 
operationally significant however. No effect was found 
for HUD raster type. 

Mental Workload . Mental workload was assessed after 
each experimental run, using the AFFTC workload 
estimate tool, and post-test, using SWORD. Pilots 
reported significantly lower post-run mental workload 
ratings for the combination of fused raster type and 
tunnel on the HUD. No significant differences were 
found for the PNF-AD configurations. Post-test, pilots 
reported no workload differences for the HUD concepts 
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yet significantly lower workload ratings for the fused 
raster with symbology PNF-AD condition. 

Situation Awareness . Situation awareness was assessed 
after each experimental run, using SART, and post-test, 
using SA-SWORD in both PF and PNF roles. Pilots 
reported significantly higher post-run ratings when the 
tunnel was paired with fusion raster type on the HUD. 
No significant differences for symbology and raster type 
was found on the PNF-AD. The post-test results 
mirrored those of the post-run data for the HUD 
concepts. Post-test, the pilots ranked the Fused- 
Symbology PNF-AD concept as having significantly 
higher SA than the other concepts. 

Rare-Event Results 

Runway Incursions . The runway incursions were made 
by a baggage cart and a fire truck. Both items were 
positioned in the same location, approximately 850 ft 
from the RN016R landing threshold and just slightly 
offset from the centerline. The weather on the runway 
incursions was held constant at 2400 ft runway visual 
range (RVR) OTW with the lowest cloud layer at 500 ft 
AFL. The FLIR visibility was very good in this 
condition - approximately 4 times the OTW RVR. 

The baggage cart runway incursion was always 
performed before the fire truck incursion. The baggage 
cart was much more difficult to see due to its small size. 
This ordering tested for “just noticeable differences” for 
runway incursion detection. For the 12 flight crews, 
only one crew saw the baggage cart (observed OTW by 
the PNF), but all 12 crews saw the fire truck. Eleven 
crews saw the fire truck OTW (7 by the PNF, 3 by the 
PF, and 1 simultaneously by the PF and PNF) and one 
crew saw it on the PNF-AD. Upon seeing the 
incursions, all crews initiated a go-around (all lower 
than 50 feet AGL). 

Navigation Error . The navigation errors were either a 
50 foot or 75 foot lateral offset (Figure 5). The offsets 
could be detected by either the PF or the PNF. The 
errors were noticeable from one of several principle 
ways (depending upon the display configuration): 

• By a disagreement between the lateral path error 
and the localizer deviation symbology (HUD and 
PNF-AD with symbology). 

• By a non-zero localizer deviation on the PFD when 
the PF is flying on the final approach path 
centerline. 

• By differences between the SV and the EV 
registration using the PNF-AD Fusion controls. 


• By differences between the runway outline and the 
EV imagery of the runway (HUD and PNF-AD). 

• By differences in the pitch/roll guidance symbol and 
the EV imagery (PF-HUD and PNF-AD) 



Figure 5. 75 foot localizer offset 


The majority of flight crews verbally noted the presence 
of the 50 foot offset (15/24) and 75 foot offset (22/24) 
during the approach. None of the pilots executed a go- 
around with this anomaly. Each performed a lateral 
correction and landed near the runway centerline. Video 
analysis showed that navigation errors were 
predominately noted by the PF (-85%) when they noticed 
that the pitch/roll guidance symbol was leading them to 
the left or right of the runway. One person (flying as the 
PNF) noted the non-zero localizer deviation on the PFD 
presentation while tracking the path centerline. 

Illegal Landings 

Each flight crew was confronted with four trials where 
weather conditions obscured the visual cues required to 
complete the landing from 100 ft HAT as defined by 
FAR §91.175. Of the 48 “illegal landing” rare event 
trials, only during six of these trials did pilots continue 
and land the aircraft. (See Table 1.) 

Discussion 

Runway Incursions 

The incurring vehicles were visible in the PNF-AD and 
HUD, yet the data suggests that EV on the HUD and 
PNF-AD were not useful for RI detection. In the HUD, 
the incurring vehicles were largely occluded by 
symbology on the HUD (flight path marker and guidance 
cue) and the small size and relatively low resolution of 
HUD made vehicle detection extremely difficult for the 
PF. 

In contrast, the vehicles were much more apparent in the 
PNF-AD. Again, the vehicle size and contrast to the 
surroundings made detection on the PNF-AD moderately 
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difficult above 200 ft AFL, particularly if the PNF only 
used cursory looks at the PNF- AD. The presence of 
symbology on the PNF- AD could also obscure the 
vehicles. Below 200 ft AFL, the vehicles were much 
more obvious in the image, but the PNF was head-out 
the vast majority of the time, ranging from 86% to 100% 
of the total time below 200 ft. AFL. 

The display concepts tested in this experiment - typical 
of current and future PF HUD and PNF-AD displays - 
showed poor incursion detection functionality. Only 
one of the runway incursion scenarios was detected 
through use of the cockpit displays. Therefore, 
requirements for display and sensor technology for 
runway incursion detection should be developed. The 
requirements for this function should span the breadth of 
the problem, including human perception, sensor design 
and detection theory, crew procedures, and crew 
interface issues. 


Table 1 . Illegal Landing Trials 


Crew 

Observation 

HUD 

AD 

3 

Go-around 

considered 

FLIR 
No Tunnel 

Fused 
No Symb. 

4 

Threshold lights 
called at 80 ft AFL 

FLIR 
No Tunnel 

FLIR/ 

Symbology 

4 

Threshold lights 
called at 60 ft AFL 

FLIR 
No Tunnel 

Fused 

Symbology 

7 

Threshold lights 
called at 100 ft AFL 

FLIR 

Tunnel 

Fused 

Symbology 

8 

Threshold lights 
called at 120 ft AFL 

FLIR 
No Tunnel 

Fused 
No Symb. 

11 

Threshold lights 
called at 90 ft AFL 

FLIR 
No Tunnel 

FLIR 
No Symb. 


Navigation Error 

The flight crews were not instructed on the course of 
action to take when confronted with a navigation error, 
and these pilots had relatively little training and 
experience with the system. Despite this, the study 
showed that lateral navigation errors were verbally 
acknowledged a significant percentage of time and, even 
when unrecognized (i.e., not explicitly verbalized), all 
flight crews landed safely and accurately on the runway. 
These results suggest that dissociations between raw 
data, sensor, and/or database presentation should be 
easily recognized and managed by experienced pilots. 
Pilot training to recognize these discrepancies could 
further improve operations in the event of this anomaly. 

Illegal Landings 

The results demonstrated the potential for flight crews to 
continue approaches to a landing during visibility 


conditions that instead require a go-around under the new 
91.175 operational rules for EFVS. On each of the six 
illegal landings, the pilot flying had excellent visibility of 
the runway using the FLIR on the HUD. However, the 
91.175 rule requires visual acquisition of the runway 
references without use of the EFVS. No effect of 
symbology type on the HUD was observed. The 
operational procedures necessary to follow the 91.175 
regulation was found to be awkward for the PF, requiring 
the PF to declutter the HUD or look-around the HUD 
combiner. The radio altitude shown on the HUD could 
be used forjudging HAT. 

For the PNF, conformance to the 91.175 required visual 
references was not influenced by the PNF-AD 
configuration. The PNF had to go head-down to read the 
altitude on the PFD or PNF-AD. The experiment did not 
use a “100 ft” AFL call-out. 

The few occurrences of “below minimums” landings 
suggest that the current regulations can be operationally 
viable. However, an aural call-out at 100 ft AFL may 
help overcome the lack of awareness to the HAT. 
Nonetheless, there still exists an awkwardness in the 
transition from EV/HUD-to-visual runway references. 
The PFs typically commented that the EFVS provided 
suitable visual references to complete the flare and 
landing. 
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